Reinforced Multi-Teacher Selection for Knowledge Distillation

In natural language processing (NLP) tasks, slow inference speed and huge footprints in GPU usage remain the bottleneck of applying pre-trained deep models production. As a popular method for model compression, knowledge distillation transfers from one or multiple large (teacher) to small (student) model. When teacher are available distillation, state-of-the-art methods assign fixed weight whole distillation. Furthermore, most existing allocate an equal every this paper, we observe that, due complexity training examples differences student capability, learning differentially can lead better performance distilled. We systematically develop reinforced dynamically weights different instances optimize Our extensive experimental results on several NLP tasks clearly verify feasibility effectiveness our approach.